Add native fetch() polyfill#188
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a native fetch() polyfill implemented in C++ on top of the existing UrlLib transports (matching the XMLHttpRequest approach), wires it behind a new CMake option, and extends the unit test suite to validate core fetch/Response behaviors.
Changes:
- Introduce
Polyfills/Fetchlibrary implementingfetch(input, init)and a minimalResponse/Headerssurface. - Wire
JSRUNTIMEHOST_POLYFILL_FETCH(default ON) and ensureUrlLibis fetched when either XHR or Fetch is enabled. - Add Mocha unit tests and a JSON asset to validate response properties and body readers (
text/arrayBuffer/json/blob) plus cloning and header behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| Tests/UnitTests/Shared/Shared.cpp | Initializes the new Fetch polyfill in the UnitTests runtime. |
| Tests/UnitTests/Scripts/tests.ts | Adds a new describe("fetch") suite covering status/ok, statusText, body readers, headers, clone, init.method, and no-arg rejection. |
| Tests/UnitTests/CMakeLists.txt | Links the new Fetch target into the UnitTests executable; Assets are globbed/copied so sample.json is picked up. |
| Tests/UnitTests/Assets/sample.json | Adds deterministic JSON payload for the response.json() test. |
| Polyfills/Fetch/Source/Fetch.h | Declares internal Fetch initializer. |
| Polyfills/Fetch/Source/Fetch.cpp | Implements fetch(), Response-like object, and headers helpers over UrlLib. |
| Polyfills/Fetch/Readme.md | Documents supported surface area and known limitations. |
| Polyfills/Fetch/Include/Babylon/Polyfills/Fetch.h | Adds public API header for initializing the polyfill. |
| Polyfills/Fetch/CMakeLists.txt | Defines the Fetch library target and links against JsRuntime, arcana, and UrlLib. |
| Polyfills/CMakeLists.txt | Adds conditional inclusion of the Fetch subdirectory. |
| CMakeLists.txt | Adds JSRUNTIMEHOST_POLYFILL_FETCH option and updates UrlLib dependency gating. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Addressed both review comments in ebe6a80:
This should turn the failing JSC/JSI/Android/Ubuntu/macOS/iOS builds green (they were all failing solely on the blob() test). |
|
CI is now fully green (23/23 ✅). Pushed four follow-up fixes after the initial submission, all surfaced by platform-specific CI:
Ready for re-review. |
Implements the WHATWG fetch() API as a native polyfill under Polyfills/Fetch/, mirroring the XMLHttpRequest polyfill layout. Like XMLHttpRequest, it is built on top of the platform-specific transports in UrlLib, so libcurl/WinHTTP/etc. behavior is identical between the two. fetch(input, init) returns a Promise that resolves to a Response-like object exposing ok/status/statusText/url/redirected/type/bodyUsed, a case-insensitive headers accessor (get/has/forEach), the body accessors text()/arrayBuffer()/json()/blob(), and clone(). Per the fetch spec, the promise only rejects on transport-level failures; a completed request with a non-2xx status (e.g. 404) still resolves with ok === false. The body is fully buffered before the promise resolves, so the body accessors may be called more than once (a deliberate lenient deviation from the spec's single-use semantics). Methods are limited to GET/POST and request bodies to strings, matching the underlying UrlLib transport. Wired behind the JSRUNTIMEHOST_POLYFILL_FETCH option (UrlLib is now fetched when either XMLHttpRequest or Fetch is enabled) and covered by new unit tests. Closes BabylonJS#98. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- blob() now detects the Blob polyfill via IsUndefined()/IsNull() instead of IsFunction(). Some JavaScriptCore/JSI builds classify constructor functions as typeof 'object', so IsFunction() incorrectly rejected even when the Blob polyfill was installed, failing the blob() test across all JSC/JSI CI jobs (matches the existing File polyfill workaround). - Add <cctype> for std::tolower/std::toupper used by ToLower/StatusText, which were relying on transitive includes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
arcana's task::then() captures the scheduler by reference and invokes it on the UrlLib worker thread when the request completes, which happens after the synchronous fetch() call has already returned. The previous stack-local JsRuntimeScheduler was therefore destroyed before the continuation ran, leaving arcana with a dangling reference. On Windows/Chakra this happened to survive, but on clang/libc++, JSC, JSI and Android it dereferenced freed memory and aborted with 'std::system_error: Invalid argument' inside JsRuntime::Dispatch (caught by ASAN as a stack-use-after-return). Heap-allocate the scheduler in a shared_ptr and co-own it from the continuation callable so it stays alive until the request completes, mirroring how XMLHttpRequest keeps its scheduler alive as a member. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Node-API-JSI declares Napi::Object::Set / Napi::Array::Set as non-const member functions, so calling Set() on 'const auto' instances failed to compile (C2663) on the *_JSI and Android targets while compiling fine under Chakra/V8. Declare the blob() 'parts' array and 'options' object as non-const, matching the other builder objects in this file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Android UnitTests app uses its own CMakeLists with an explicit polyfill link list, separate from the desktop Tests/UnitTests target. Without Fetch there, Shared.cpp failed to find <Babylon/Polyfills/Fetch.h> on Android_JSC and Android_V8. Add 'PRIVATE Fetch' to mirror the desktop test target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1118799 to
fbe94db
Compare
| request->SendAsync().then(*scheduler, arcana::cancellation::none(), | ||
| [deferred, request, env, scheduler](const arcana::expected<void, std::exception_ptr>& result) { | ||
| const int status = static_cast<int>(request->StatusCode()); | ||
|
|
||
| // Per the WHATWG fetch spec, only transport-level failures reject. A completed | ||
| // request with a non-2xx status (e.g. 404) still resolves with response.ok === false. | ||
| // A status of 0 indicates the transport never produced a response (network error). | ||
| if (result.has_error() || status == 0) | ||
| { | ||
| deferred.Reject(Napi::Error::New(env, "fetch: network request failed").Value()); | ||
| return; | ||
| } | ||
|
|
||
| auto data = std::make_shared<ResponseData>(); | ||
| data->statusCode = status; | ||
| data->url = std::string{request->ResponseUrl()}; | ||
| for (const auto& header : request->GetAllResponseHeaders()) | ||
| { | ||
| data->headers.emplace_back(header.first, header.second); | ||
| } | ||
| const auto responseBuffer = request->ResponseBuffer(); | ||
| data->body = std::make_shared<std::vector<std::byte>>(responseBuffer.begin(), responseBuffer.end()); | ||
|
|
||
| deferred.Resolve(BuildResponse(env, data)); | ||
| }); |
There was a problem hiding this comment.
A throw here (e.g. from BuildResponse) is captured by arcana into the discarded result task and dropped, so the promise never settles and await fetch(...) hangs.
| request->SendAsync().then(*scheduler, arcana::cancellation::none(), | |
| [deferred, request, env, scheduler](const arcana::expected<void, std::exception_ptr>& result) { | |
| const int status = static_cast<int>(request->StatusCode()); | |
| // Per the WHATWG fetch spec, only transport-level failures reject. A completed | |
| // request with a non-2xx status (e.g. 404) still resolves with response.ok === false. | |
| // A status of 0 indicates the transport never produced a response (network error). | |
| if (result.has_error() || status == 0) | |
| { | |
| deferred.Reject(Napi::Error::New(env, "fetch: network request failed").Value()); | |
| return; | |
| } | |
| auto data = std::make_shared<ResponseData>(); | |
| data->statusCode = status; | |
| data->url = std::string{request->ResponseUrl()}; | |
| for (const auto& header : request->GetAllResponseHeaders()) | |
| { | |
| data->headers.emplace_back(header.first, header.second); | |
| } | |
| const auto responseBuffer = request->ResponseBuffer(); | |
| data->body = std::make_shared<std::vector<std::byte>>(responseBuffer.begin(), responseBuffer.end()); | |
| deferred.Resolve(BuildResponse(env, data)); | |
| }); | |
| request->SendAsync() | |
| .then(*scheduler, arcana::cancellation::none(), | |
| [deferred, request, env](const arcana::expected<void, std::exception_ptr>& result) { | |
| const int status = static_cast<int>(request->StatusCode()); | |
| // Per the WHATWG fetch spec, only transport-level failures reject. A completed | |
| // request with a non-2xx status (e.g. 404) still resolves with response.ok === false. | |
| // A status of 0 indicates the transport never produced a response (network error). | |
| if (result.has_error() || status == 0) | |
| { | |
| throw std::runtime_error{"fetch: network request failed"}; | |
| } | |
| auto data = std::make_shared<ResponseData>(); | |
| data->statusCode = status; | |
| data->url = std::string{request->ResponseUrl()}; | |
| for (const auto& header : request->GetAllResponseHeaders()) | |
| { | |
| data->headers.emplace_back(header.first, header.second); | |
| } | |
| const auto responseBuffer = request->ResponseBuffer(); | |
| data->body = std::make_shared<std::vector<std::byte>>(responseBuffer.begin(), responseBuffer.end()); | |
| deferred.Resolve(BuildResponse(env, data)); | |
| }) | |
| .then(*scheduler, arcana::cancellation::none(), | |
| [deferred, env, scheduler](const arcana::expected<void, std::exception_ptr>& result) { | |
| if (result.has_error()) | |
| { | |
| deferred.Reject(Napi::Error::New(env, result.error()).Value()); | |
| } | |
| }); |
There was a problem hiding this comment.
Fixed in ac2019f — split into two chained .then() continuations: the first does the work and resolves, the second inspects the expected result and surfaces any error (including a throw from BuildResponse) as a promise rejection. The scheduler is co-owned by the second continuation so it outlives the in-flight request. Good catch — this would have hung await fetch(...) on any post-completion throw.
| catch (const Napi::Error& error) | ||
| { | ||
| deferred.Reject(error.Value()); | ||
| } | ||
| catch (const std::exception& error) | ||
| { | ||
| deferred.Reject(Napi::Error::New(env, std::string{"fetch: "} + error.what()).Value()); | ||
| } |
There was a problem hiding this comment.
Optional: a single catch (...) via the exception_ptr overload also removes the Napi::Error-vs-std::exception prefix asymmetry.
| catch (const Napi::Error& error) | |
| { | |
| deferred.Reject(error.Value()); | |
| } | |
| catch (const std::exception& error) | |
| { | |
| deferred.Reject(Napi::Error::New(env, std::string{"fetch: "} + error.what()).Value()); | |
| } | |
| catch (...) | |
| { | |
| deferred.Reject(Napi::Error::New(env, std::current_exception()).Value()); | |
| } |
There was a problem hiding this comment.
Done — collapsed the two catch blocks into a single catch (...) using the exception_ptr Error overload, removing the Napi::Error-vs-std::exception prefix asymmetry.
| const std::string upper = [&]() { | ||
| std::string result = method; | ||
| std::transform(result.begin(), result.end(), result.begin(), [](unsigned char c) { return static_cast<char>(std::toupper(c)); }); | ||
| return result; | ||
| }(); | ||
|
|
||
| if (upper == "GET") | ||
| { | ||
| return UrlLib::UrlMethod::Get; | ||
| } | ||
| if (upper == "POST") | ||
| { | ||
| return UrlLib::UrlMethod::Post; | ||
| } |
There was a problem hiding this comment.
Drop the inline toupper and compare with EqualsIgnoreCase, which also removes the second case-fold:
| const std::string upper = [&]() { | |
| std::string result = method; | |
| std::transform(result.begin(), result.end(), result.begin(), [](unsigned char c) { return static_cast<char>(std::toupper(c)); }); | |
| return result; | |
| }(); | |
| if (upper == "GET") | |
| { | |
| return UrlLib::UrlMethod::Get; | |
| } | |
| if (upper == "POST") | |
| { | |
| return UrlLib::UrlMethod::Post; | |
| } | |
| if (EqualsIgnoreCase(method, "GET")) | |
| { | |
| return UrlLib::UrlMethod::Get; | |
| } | |
| if (EqualsIgnoreCase(method, "POST")) | |
| { | |
| return UrlLib::UrlMethod::Post; | |
| } |
Duplicates XHR's MethodType::StringToEnum (which is case-sensitive, so non-conforming). Since UrlLib owns UrlMethod, this mapping ideally belongs there — follow-up, not this PR.
There was a problem hiding this comment.
Done — ParseMethod now uses EqualsIgnoreCase. Agreed the GET/POST → UrlMethod mapping ideally belongs in UrlLib (which owns UrlMethod); leaving that as a follow-up as you suggested.
| const char* StatusText(int statusCode) | ||
| { | ||
| switch (statusCode) | ||
| { | ||
| case 200: return "OK"; | ||
| case 201: return "Created"; | ||
| case 202: return "Accepted"; | ||
| case 204: return "No Content"; | ||
| case 206: return "Partial Content"; | ||
| case 301: return "Moved Permanently"; | ||
| case 302: return "Found"; | ||
| case 304: return "Not Modified"; | ||
| case 307: return "Temporary Redirect"; | ||
| case 308: return "Permanent Redirect"; | ||
| case 400: return "Bad Request"; | ||
| case 401: return "Unauthorized"; | ||
| case 403: return "Forbidden"; | ||
| case 404: return "Not Found"; | ||
| case 405: return "Method Not Allowed"; | ||
| case 408: return "Request Timeout"; | ||
| case 409: return "Conflict"; | ||
| case 429: return "Too Many Requests"; | ||
| case 500: return "Internal Server Error"; | ||
| case 502: return "Bad Gateway"; | ||
| case 503: return "Service Unavailable"; | ||
| case 504: return "Gateway Timeout"; | ||
| default: return ""; | ||
| } | ||
| } |
There was a problem hiding this comment.
statusText is a workaround for UrlLib exposing only the numeric code (returns "" for any unlisted code). Belongs in UrlLib — expose the real reason phrase, falling back to a code→text table for HTTP/2+ where the wire carries none. Would also give XHR, currently missing statusText, one for free. Follow-up, not this PR.
There was a problem hiding this comment.
Agreed — the statusText table is a workaround for UrlLib exposing only the numeric code. The real fix belongs in UrlLib (expose the reason phrase, falling back to a code→text table for HTTP/2+ where the wire carries none), which would also give XHR a statusText for free. Tracked as a follow-up in #193 (not this PR).
| int statusCode{}; | ||
| std::string url; | ||
| std::vector<std::pair<std::string, std::string>> headers; | ||
| std::shared_ptr<std::vector<std::byte>> body; |
There was a problem hiding this comment.
body doesn't need to be a shared_ptr. ResponseData is only ever held as a shared_ptr<ResponseData>, and every accessor plus clone() captures that same pointer — so body is already shared and kept alive through the enclosing struct. The inner shared_ptr<vector<byte>> adds an allocation and an indirection without changing sharing, and misleads the reader into thinking clones get independent bodies (they don't — clone() reuses the same ResponseData). Make it a plain std::vector<std::byte> body; and change the data->body-> use sites to data->body..
There was a problem hiding this comment.
Done — ResponseData::body is now a plain std::vector<std::byte> and the accessor/clone use sites were updated to data->body.. You're right that the inner shared_ptr only added an allocation/indirection and falsely implied clones get independent bodies.
| std::string ToLower(std::string value) | ||
| { | ||
| std::transform(value.begin(), value.end(), value.begin(), [](unsigned char c) { return static_cast<char>(std::tolower(c)); }); | ||
| return value; | ||
| } |
There was a problem hiding this comment.
With non-allocating compares, ToLower isn't needed anywhere — replace it with a case-insensitive comparator used by both ParseMethod and FindHeader:
| std::string ToLower(std::string value) | |
| { | |
| std::transform(value.begin(), value.end(), value.begin(), [](unsigned char c) { return static_cast<char>(std::tolower(c)); }); | |
| return value; | |
| } | |
| bool EqualsIgnoreCase(std::string_view a, std::string_view b) | |
| { | |
| return std::equal(a.begin(), a.end(), b.begin(), b.end(), [](unsigned char l, unsigned char r) { | |
| return std::tolower(l) == std::tolower(r); | |
| }); | |
| } |
Coordinated with the ParseMethod and FindHeader suggestions — batch all three. (<string_view> comes in transitively via UrlLib.h.)
There was a problem hiding this comment.
Done — added the non-allocating EqualsIgnoreCase(string_view, string_view) comparator and removed ToLower; it's now used by both ParseMethod and FindHeader. (Added an explicit #include <string_view> rather than rely on the transitive include from UrlLib.h.)
| std::optional<std::string> FindHeader(const ResponseData& data, const std::string& name) | ||
| { | ||
| const std::string lowerName = ToLower(name); | ||
| for (const auto& header : data.headers) | ||
| { | ||
| if (ToLower(header.first) == lowerName) | ||
| { | ||
| return header.second; | ||
| } | ||
| } | ||
| return std::nullopt; | ||
| } |
There was a problem hiding this comment.
Each get/has allocates lowerName plus a ToLower temporary for every header scanned. Use the non-allocating comparator instead:
| std::optional<std::string> FindHeader(const ResponseData& data, const std::string& name) | |
| { | |
| const std::string lowerName = ToLower(name); | |
| for (const auto& header : data.headers) | |
| { | |
| if (ToLower(header.first) == lowerName) | |
| { | |
| return header.second; | |
| } | |
| } | |
| return std::nullopt; | |
| } | |
| std::optional<std::string> FindHeader(const ResponseData& data, std::string_view name) | |
| { | |
| for (const auto& header : data.headers) | |
| { | |
| if (EqualsIgnoreCase(header.first, name)) | |
| { | |
| return header.second; | |
| } | |
| } | |
| return std::nullopt; | |
| } |
There was a problem hiding this comment.
Done — FindHeader now uses EqualsIgnoreCase, so no more per-header lowerName allocation while scanning.
| Napi::Object BuildResponse(Napi::Env env, const std::shared_ptr<ResponseData>& data); | ||
|
|
There was a problem hiding this comment.
Unnecessary forward declaration — BuildHeaders (the only thing before the L184 definition) doesn't call BuildResponse, and the recursive clone call is inside BuildResponse's own body where its name is already in scope. Remove it:
| Napi::Object BuildResponse(Napi::Env env, const std::shared_ptr<ResponseData>& data); |
There was a problem hiding this comment.
Done — removed the unnecessary BuildResponse forward declaration.
| // Use IsUndefined()/IsNull() rather than IsFunction() to detect the Blob | ||
| // polyfill: some JavaScriptCore/JSI builds classify constructor functions as | ||
| // typeof 'object', so napi_typeof reports napi_object and IsFunction() would | ||
| // incorrectly reject even when the Blob polyfill is installed. |
There was a problem hiding this comment.
The workaround is fine, but the root cause is a Node-API-JSC bug worth tracking: napi_typeof (js_native_api_javascriptcore.cc) decides function-ness solely via JSObjectIsFunction, which returns false for JSObjectMakeConstructor constructors on some builds (libjavascriptcoregtk) — so it reports napi_object where V8 returns napi_function. A fix there (also consult JSObjectIsConstructor) would let this and the File polyfill's identical workaround be removed. Worth a tracking issue rather than spreading per-polyfill copies.
There was a problem hiding this comment.
Agreed — root cause is the Node-API-JSC napi_typeof path deciding function-ness solely via JSObjectIsFunction, which returns false for JSObjectMakeConstructor constructors on some JSC builds, so it reports napi_object where V8 returns napi_function. The File polyfill carries the identical workaround. A fix there (also consulting JSObjectIsConstructor) would let both drop the workaround rather than spreading per-polyfill copies. Tracked in #194; keeping the workaround here for now.
…thod compares Address review feedback on the native fetch() polyfill: - Fix a hang: a throw inside the SendAsync continuation (e.g. from BuildResponse) was captured into the discarded task result and dropped, so the promise never settled and await fetch(...) hung. Split into two chained .then() continuations -- the first does the work and resolves, the second surfaces any error as a promise rejection. The scheduler is co-owned by the second continuation so it outlives the request. - Replace ToLower with a non-allocating EqualsIgnoreCase comparator used by both ParseMethod and FindHeader (removes per-header allocations). - Make ResponseData::body a plain std::vector<std::byte> instead of a shared_ptr; the struct is always held by shared_ptr, so the inner pointer only added an allocation and falsely implied clones get independent bodies. - Remove the unnecessary BuildResponse forward declaration. - Collapse the two fetch() catch blocks into a single catch (...) via the exception_ptr Error overload, removing the message-prefix asymmetry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the WHATWG etch() API as a native polyfill under
Polyfills/Fetch/, mirroring theXMLHttpRequestpolyfill layout. LikeXMLHttpRequest, it is built on the same platform-specificUrlLibtransports, so libcurl/WinHTTP/etc. behavior is identical between the two.Closes #98.
API
fetch(input, init)returns aPromisethat resolves to aResponse-like object exposing:ok,status,statusText,url,redirected,type,bodyUsedheaderswithget(name)/has(name)/forEach(cb)(case-insensitive)text(),arrayBuffer(),json(),blob()(each returns aPromise)clone()Behavior / deliberate limitations
ok === false.bodyUsedstaysfalse) — a lenient deviation from the spec's single-use semantics.GET/POSTonly and string request bodies only, matching the underlyingUrlLibtransport (same constraints asXMLHttpRequest).blob()requires theBlobpolyfill to be initialized.Wiring
JSRUNTIMEHOST_POLYFILL_FETCHoption (defaultON).UrlLibis now fetched when eitherXMLHttpRequestorFetchis enabled.describe("fetch")block (12 cases: status/ok, 404-resolves, statusText, text/arrayBuffer/json/blob, case-insensitive headers, clone, init method, no-arg rejection). A smallAssets/sample.jsonbacks the deterministicjson()test.Verification
Built the
Fetchtarget underwarnings_as_errorsand ran the full UnitTests suite on Win32/Chakra: 192 passing, including all 12 new fetch cases.Context
This is step 2 of the fetch native-polyfill pivot discussed in BabylonNative#1707 (reviewers asked fetch to follow the same native pattern as File/FileReader (#169) and AbortController). Once this lands, BabylonNative#1707 becomes a small bump that initializes
Polyfills::Fetch.